Skip to content

feat(guest-agent): collect on-demand GPU attestation evidence - #1112

Merged
kvinwang merged 8 commits into
nextfrom
feat/attest-gpu
Aug 24, 2026
Merged

feat(guest-agent): collect on-demand GPU attestation evidence#1112
kvinwang merged 8 commits into
nextfrom
feat/attest-gpu

Conversation

@kvinwang

@kvinwang kvinwang commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Problem

GpuInfo replays a record written at boot and cannot answer whether a GPU can produce fresh attestation evidence now. A driver reload can leave a device that still responds to NVML but can no longer attest.

Fix

Add AttestGpu, which collects fresh GPU evidence for a caller-chosen 32-byte nonce.

The response is vendor-neutral:

message AttestGpuResponse {
  repeated GpuEvidenceBundle bundles = 1;
}

message GpuEvidenceBundle {
  string vendor = 1;
  string format = 2;
  bytes evidence = 3;
}

The current NVIDIA implementation returns one bundle:

  • vendor: nvidia
  • format: nvidia-nvattest-collect-evidence-json-v1
  • evidence: the opaque nvattest collect-evidence evidences array

The agent only collects evidence. It does not run the local NVIDIA verifier or return an appraisal. A relying party selects a verifier using vendor and format, then verifies the signature, certificate chain, measurements, and nonce embedded in the evidence.

The response intentionally does not echo the nonce because the trusted nonce binding is inside the GPU-signed evidence. It also does not claim that the GPU is attached to this TD: current GPU evidence remains relayable until TDISP/TEE-IO provides device binding.

Evidence collection is serialized to avoid concurrent access to the same devices, but it is not rate-limited because collect-evidence does not fetch OCSP or RIM collateral from NVIDIA.

The API is exposed by the Rust, Python, Go, JavaScript, and curl SDK documentation.

Shared crate

The boot gate previously owned the NVIDIA CLI invocation, proxy validation, timeout, and nonce checking. This PR extracts that code into the nvattest crate and rewires dstack-util to use it. run_command uses kill_on_drop, so a timed-out or cancelled tool does not outlive its caller.

Verification

  • cargo check -p nvattest -p dstack-guest-agent
  • cargo test -p nvattest: 9 passed
  • cargo test -p dstack-guest-agent gpu_attest: 3 passed
  • cargo check in sdk/rust
  • go test ./dstack -run TestAttestGpu
  • cargo fmt --all --check
  • git diff --check
  • Python source compilation passed; pytest collection is blocked by the unavailable evidence_api dependency.
  • JavaScript tests were not run because dependencies are not installed in the review environment.

Copilot AI lite review requested due to automatic review settings August 24, 2026 03:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

The boot gate owned the only code that knows how to call `nvattest`: the
argument shape, the proxy-URL rules that stop a misconfigured value smuggling a
different collateral endpoint past `{proxy}/ocsp`, and the nonce check. The
guest agent is about to need all three for on-demand attestation, and a second
copy is a second set of bugs.

The crate keeps the invocation and drops the appraisal, because the boot gate
and a runtime liveness check want different answers from the same bytes:
`check_nonce` only asserts that every claim answers the nonce it was given, and
device counts, CC state and application policy stay with the caller.

`run` and `attest` are separate so the boot gate keeps persisting stdout before
it judges the exit status -- a failed appraisal is exactly when the evidence is
worth having on disk. `run_command` also gains `kill_on_drop`, so a wedged tool
no longer outlives the timeout that gave up on it.
`GpuInfo` replays a record written at boot, which cannot answer "is the GPU
working *now*". That question has a concrete failure behind it: unloading and
reloading the NVIDIA driver leaves a device that still answers NVML but can no
longer attest, and the repo's own H100 evidence run records exactly that
(`nvidia_smi_rc=9`, attestation `244`, recovered only by a full VM stop/start).
`security-model.md` already told higher-assurance deployments to "re-attest
before using a newly initialized GPU" without offering an API to do it.

`AttestGpu` runs nvattest against a caller-chosen nonce. The nonce is passed to
the GPU verbatim and must be exactly 32 bytes, which is what SPDM fixes it at:
no padding and no hashing, so a caller compares its own bytes against
`eat_nonce` rather than reversing a transform.

**What it does not do.** It is not a remote attestation claim, and both the
proto and every SDK say so at the field. An NVIDIA report binds the device and
the nonce and nothing else, so a hostile host can relay the challenge to a real
GPU elsewhere; deriving the nonce from a TDX quote does not help, because the
relay can derive it too. Only TDISP/TEE-IO device binding closes that. Remote
evidence therefore stays with the boot-time `gpu-attestation` event, which
measured code emits before any workload exists and which the event log binds to
the quote.

No runtime event is emitted for the same reason: every dstack verifier stops
scanning the event log at `system-ready`, so measuring this would add noise
that nothing reads and invite the misreading above.

Calls are serialised behind a mutex and rate-limited to one per 10s. The CVM is
a single trust domain, so a container spinning on this endpoint only hurts its
own deployment -- the reason to bound it is that each call spawns a process and
fetches OCSP and RIM collateral from NVIDIA, whose SDK cache is per-process and
so starts cold every time. A retry loop would hammer NVIDIA's services from
every CVM running the buggy app.
The response documented one reason a relying party must not trust it -- an
NVIDIA report binds the device and nonce but not the TD, so it is relayable.
There is a blunter reason that comes first, and the docs did not say it.

`nvattest --verifier local` returns the verifier's *conclusion*, not the GPU's
signed report. The repo's own captured H100 output shows the detached EAT is
`alg:none` with an empty signature, issued by `NVAT-LOCAL-VERIFIER`, and claims
such as `x-nvidia-gpu-attestation-report-signature-verified` are assertions
about a check already performed; the signed SPDM report and certificate chain
are consumed during verification and never appear in the output. A third party
handed this JSON therefore has nothing to check at all, relay or no relay.

Worth being precise about because it also explains why the boot-time path is
sound with the same unsigned artifact: that evidence is trusted not because it
self-authenticates but because measured dstack code pinned by `os_image_hash`
appraised it before any workload existed, with sha256 of the exact bytes in
RTMR3 under the quote. The appraiser is what is trusted, not the JSON.

A test pins the format claim against the fixture, so an SDK that starts signing
the EAT fails here rather than leaving the API docs quietly wrong.
@kvinwang

Copy link
Copy Markdown
Collaborator Author

Adding a correction to the framing above, from checking the actual artifact rather than reasoning from the claim names.

The PR body justifies "not remote evidence" via the relay argument. There is a blunter reason that comes first: the output is unsigned.

nvattest --verifier local returns the verifier's conclusion, not the GPU's signed report. From this repo's own captured H100 output (dstack-util/tests/fixtures/gpu_attestation_h100.json):

  • the embedded detached_eat is {"alg":"none"} with an empty signature, iss: NVAT-LOCAL-VERIFIER
  • x-nvidia-gpu-attestation-report-signature-verified: true is an assertion about a check already performed, not proof anyone can redo
  • x-nvidia-gpu-attestation-report-cert-chain holds verdicts (x-nvidia-cert-ocsp-status: "good"), not the chain

The GPU's signed SPDM report and certificate chain are consumed during verification and never appear in the output. So a third party handed this JSON has nothing to check at all — relay or no relay.

This does not change the design, but it does sharpen why it is drawn this way, and it explains something worth stating explicitly: the boot-time path is sound with the same unsigned artifact, because its trust does not come from the JSON self-authenticating. It comes from measured dstack code (pinned by os_image_hash) having appraised the GPU before any workload existed, with sha256 of the exact bytes in RTMR3 under the quote. The appraiser is what is trusted, not the artifact.

11c1e40f2b states both reasons in the proto, all four SDKs, the curl reference and the CHANGELOG, and adds a test pinning the format claim against the fixture — if a future SDK starts signing the EAT, that test fails instead of the API docs quietly becoming wrong.

If you want third-party-verifiable GPU evidence later, there are two routes, both larger than this PR: nvattest collect-evidence emits the raw signed report and cert chain for someone else to appraise, and --verifier remote returns an NVIDIA-signed EAT from NRAS. Neither gives TD binding — that still needs TDISP/TEE-IO — but either would at least let a remote party check the GPU's own signature.

Comment thread dstack/guest-agent/src/gpu_attest.rs Dismissed
Comment thread dstack/nvattest/src/lib.rs Dismissed
Comment thread dstack/nvattest/src/lib.rs Dismissed
Comment thread dstack/nvattest/src/lib.rs Dismissed
Comment thread sdk/rust/tests/test_client.rs Dismissed
Comment thread sdk/rust/tests/test_client.rs Dismissed
`AttestGpu` returned only `nvattest attest --verifier local` output, which is
the verifier's conclusion. That conclusion is unsigned -- its detached EAT is
`alg:none` from `NVAT-LOCAL-VERIFIER` -- so a third party handed it had nothing
to check, and the API could only ever be a local health check.

`collect-evidence` emits what the GPU actually signed: per device, the base64
SPDM attestation report and its certificate chain, over the caller's nonce. The
response now carries that as `evidence`, so a relying party can verify the chain
to NVIDIA's root, check the report signature, confirm the nonce inside the
report, and compare measurements against NVIDIA's RIM documents -- with its own
verifier, trusting nothing this CVM says.

The local verdict is still useful to a caller inside the CVM, which is in the
agent's trust domain and usually just wants the answer, so it stays as
`appraisal`. Splitting the two is the RATS distinction: evidence is what the
attester produces, appraisal is a verifier's opinion about it, and only the
first travels.

The agent collects once and appraises those exact bytes via
`--gpu-evidence-source=file` rather than running two independent attestations,
so the two halves provably describe the same report. The SDK independently
rejects an evidence file whose nonce does not match the appraisal nonce.

This does not change what the evidence proves about *placement*: an NVIDIA
report binds the device and the nonce and nothing else, so it stays relayable
from a genuine remote GPU until TDISP/TEE-IO. What changes is that the part a
third party can check is now actually reaching them.
Comment thread dstack/nvattest/src/lib.rs Dismissed
Comment thread dstack/nvattest/src/lib.rs Fixed
@kvinwang kvinwang changed the title feat(guest-agent): add AttestGpu for on-demand GPU attestation feat(guest-agent): collect on-demand GPU attestation evidence Aug 24, 2026
# Conflicts:
#	CHANGELOG.md
#	docs/attestation-tdx.md
#	dstack/guest-agent/src/rpc_service.rs
#	sdk/curl/api.md
#	sdk/go/README.md
#	sdk/go/dstack/client_test.go
#	sdk/js/README.md
#	sdk/python/README.md
#	sdk/python/tests/test_client.py
@kvinwang
kvinwang merged commit 38f544a into next Aug 24, 2026
15 checks passed
@kvinwang
kvinwang deleted the feat/attest-gpu branch August 24, 2026 07:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants